{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "# Vertex AI Model Builder SDK: AutoML Tabular Training and Prediction\n",
    "\n",
    "## Overview\n",
    "\n",
    "In this notebook, you learn how to use the Vertex AI Python client library to train and deploy a tabular classification model for online prediction.\n",
    "\n",
    "## Learning Objective\n",
    "\n",
    "In this notebook, you learn how to:\n",
    "\n",
    "* Create a Vertex AI model training job.\n",
    "* Train an AutoML tabular model.\n",
    "* Deploy the `model` resource to a serving `endpoint` resource.\n",
    "* Make a prediction by sending data.\n",
    "* Undeploy the `model` resource.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, you will use Vertex AI Python client library to train and make predictions on an AutoML model based on a tabular dataset. Alternatively, you can train and make predictions on models by using the gcloud command-line tool or by using the online Cloud Console.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/automl-tabular-classification.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n",
    "\n",
    "**Make sure to enable the Vertex AI API and Compute Engine API.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "install_aip"
   },
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "PyQmSRbKA8r-"
   },
   "outputs": [],
   "source": [
    "# Setup your dependencies\n",
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "USER_FLAG = \"\"\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b03b7f4487ff"
   },
   "source": [
    "Install the latest version of the Vertex AI client library.\n",
    "\n",
    "Run the following command in your virtual environment to install the Vertex SDK for Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "d489d38261dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.7/site-packages (1.1.1)\n",
      "Collecting google-cloud-aiplatform\n",
      "  Downloading google_cloud_aiplatform-1.3.0-py2.py3-none-any.whl (1.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.3 MB 7.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.19.0)\n",
      "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.23.2)\n",
      "Requirement already satisfied: google-api-core[grpc]<3.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.31.1)\n",
      "Requirement already satisfied: google-cloud-storage<2.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.41.1)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (21.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.53.0)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (3.16.0)\n",
      "Requirement already satisfied: six>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.16.0)\n",
      "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (2021.1)\n",
      "Requirement already satisfied: google-auth<2.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.34.0)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (2.25.1)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (49.6.0.post20210108)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.29.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.38.1)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (4.7.2)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (4.2.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (0.2.7)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.7.2)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.1.2)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.14.6)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.20)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-aiplatform) (2.4.7)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (0.4.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (2021.5.30)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (1.26.6)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (4.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform) (2.10)\n",
      "Installing collected packages: google-cloud-aiplatform\n",
      "\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "Successfully installed google-cloud-aiplatform-1.3.0\n"
     ]
    }
   ],
   "source": [
    "# Upgrade the specified package to the newest available version\n",
    "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "install_storage"
   },
   "source": [
    "Install the Cloud Storage library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "qssss-KSlugo"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-storage in /opt/conda/lib/python3.7/site-packages (1.41.1)\n",
      "Collecting google-cloud-storage\n",
      "  Downloading google_cloud_storage-1.42.0-py2.py3-none-any.whl (105 kB)\n",
      "\u001b[K     |████████████████████████████████| 105 kB 8.1 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.3.2)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.34.0)\n",
      "Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.7.2)\n",
      "Requirement already satisfied: google-api-core<3.0dev,>=1.29.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (1.31.1)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (2.25.1)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (21.0)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (3.16.0)\n",
      "Requirement already satisfied: six>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (1.16.0)\n",
      "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (2021.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (1.53.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (49.6.0.post20210108)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.2.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.2.7)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.7.2)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (1.1.2)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (1.14.6)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage) (2.20)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (2.4.7)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.4.8)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (1.26.6)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (2.10)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (2021.5.30)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (4.0.0)\n",
      "Installing collected packages: google-cloud-storage\n",
      "Successfully installed google-cloud-storage-1.42.0\n"
     ]
    }
   ],
   "source": [
    "# Upgrade the specified package to the newest available version\n",
    "! pip install {USER_FLAG} --upgrade google-cloud-storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d3a26cb9b19d"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-04-c846b6079446\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "PROJECT_ID = \"\"\n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJYoRfYng0XZ"
   },
   "source": [
    "Otherwise, set your project ID here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "riG_qUokg0XZ"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"qwiklabs-gcp-04-c846b6079446\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06571eb4063b"
   },
   "source": [
    "### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "697568e92bd6"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "from datetime import datetime\n",
    "\n",
    "# Use a timestamp to ensure unique resources\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "This notebook demonstrates how to use Model Builder SDK to create an AutoML model based on a tabular dataset. You will need to provide a Cloud Storage bucket where the dataset will be stored.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all of your \n",
    "Cloud Storage buckets.\n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
    "available](https://cloud.google.com/vertex-ai/docs/general/locations). You may\n",
    "not use a Multi-Regional Storage bucket for training with Vertex AI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"gs://qwiklabs-gcp-04-c846b6079446\"  # @param {type:\"string\"}\n",
    "REGION = \"us-central1\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "cf221059d072"
   },
   "outputs": [],
   "source": [
    "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://qwiklabs-gcp-04-c846b6079446\":\n",
    "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-04-c846b6079446aip-20210826051658/...\n"
     ]
    }
   ],
   "source": [
    "! gcloud storage buckets create --location $REGION $BUCKET_NAME"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "! gcloud storage ls --all-versions --long $BUCKET_NAME"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d89a66b8923f"
   },
   "source": [
    "### Copy dataset into your Cloud Storage bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "59a46204cddb"
   },
   "outputs": [],
   "source": [
    "IMPORT_FILE = \"petfinder-tabular-classification_toy.csv\"\n",
    "! gcloud storage cp gs://cloud-training/mlongcp/v3.0_MLonGC/pdtrust_toy_datasets/{IMPORT_FILE} {BUCKET_NAME}/data/\n",    "\n",
    "gcs_source = f\"{BUCKET_NAME}/data/{IMPORT_FILE}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y9Uo3tifg1kx"
   },
   "source": [
    "### Import Vertex SDK for Python\n",
    "\n",
    "Import the Vertex SDK into your Python environment and initialize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "pRUOFELefqf1"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "import os\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "643dfd86b00d"
   },
   "source": [
    "## Tutorial\n",
    "\n",
    "Now you are ready to create your AutoML Tabular model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8f4f50a0112c"
   },
   "source": [
    "### Create a Managed Tabular Dataset from a CSV\n",
    "\n",
    "This section will create a dataset from a CSV file stored on your GCS bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "f1eef64ee47b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.datasets.dataset:Creating TabularDataset\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:Create TabularDataset backing LRO: projects/1075205415941/locations/us-central1/datasets/1945247175768276992/operations/1110822578768838656\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:TabularDataset created. Resource name: projects/1075205415941/locations/us-central1/datasets/1945247175768276992\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:To use this TabularDataset in another session:\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:ds = aiplatform.TabularDataset('projects/1075205415941/locations/us-central1/datasets/1945247175768276992')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'projects/1075205415941/locations/us-central1/datasets/1945247175768276992'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = dataset = aiplatform.TabularDataset.create(\n",
    "    display_name=\"petfinder-tabular-dataset\",\n",
    "    gcs_source=gcs_source,\n",
    ")\n",
    "\n",
    "ds.resource_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ba5011d50ac7"
   },
   "source": [
    "### Launch a Training Job to Create a Model\n",
    "\n",
    "Once we have defined your training script, we will create a model. The `run` function creates a training pipeline that trains and creates a `Model` object. After the training pipeline completes, the `run` function returns the `Model` object.\n",
    "\n",
    "**NOTE: It takes nearly 2 hours 15 minutes to complete the training. Please wait till the training get completed.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "24c2c081d683"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:16: DeprecationWarning: consider using column_specs instead. column_transformations will be deprecated in the future.\n",
      "  app.launch_new_instance()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.training_jobs:View Training:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/1715908841423503360?project=1075205415941\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.training_jobs:AutoMLTabularTrainingJob run completed. Resource name: projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360\n",
      "INFO:google.cloud.aiplatform.training_jobs:Model available at projects/1075205415941/locations/us-central1/models/3676687718445744128\n"
     ]
    }
   ],
   "source": [
    "# TODO 1\n",
    "# Constructs a AutoML Tabular Training Job\n",
    "job = aiplatform.AutoMLTabularTrainingJob(\n",
    "    display_name=\"train-petfinder-automl-1\",\n",
    "    optimization_prediction_type=\"classification\",\n",
    "    column_transformations=[\n",
    "        {\"categorical\": {\"column_name\": \"Type\"}},\n",
    "        {\"numeric\": {\"column_name\": \"Age\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Breed1\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Color1\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Color2\"}},\n",
    "        {\"categorical\": {\"column_name\": \"MaturitySize\"}},\n",
    "        {\"categorical\": {\"column_name\": \"FurLength\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Vaccinated\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Sterilized\"}},\n",
    "        {\"categorical\": {\"column_name\": \"Health\"}},\n",
    "        {\"numeric\": {\"column_name\": \"Fee\"}},\n",
    "        {\"numeric\": {\"column_name\": \"PhotoAmt\"}},\n",
    "    ],\n",
    ")\n",
    "\n",
    "# TODO 2a\n",
    "# Create and train the model object\n",
    "# This will take around two hour and half to run\n",
    "model = job.run(\n",
    "    dataset=ds,\n",
    "    target_column=\"Adopted\",\n",
    "    # TODO 2b\n",
    "    # Define training, validation and test fraction for training\n",
    "    training_fraction_split=0.8,\n",
    "    validation_fraction_split=0.1,\n",
    "    test_fraction_split=0.1,\n",
    "    model_display_name=\"adopted-prediction-model\",\n",
    "    disable_early_stopping=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "93a4d034c73b"
   },
   "source": [
    "### Deploy your model\n",
    "\n",
    "Before you use your model to make predictions, you need to deploy it to an `Endpoint`. You can do this by calling the `deploy` function on the `Model` resource. This function does two things:\n",
    "\n",
    "1. Creates an `Endpoint` resource to which the `Model` resource will be deployed.\n",
    "2. Deploys the `Model` resource to the `Endpoint` resource.\n",
    "\n",
    "Deploy your model.\n",
    "\n",
    "### NOTE: Wait until the model **FINISHES** deployment before proceeding to prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "a371544057d9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.models:Creating Endpoint\n",
      "INFO:google.cloud.aiplatform.models:Create Endpoint backing LRO: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936/operations/7965582686603444224\n",
      "INFO:google.cloud.aiplatform.models:Endpoint created. Resource name: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n",
      "INFO:google.cloud.aiplatform.models:To use this Endpoint in another session:\n",
      "INFO:google.cloud.aiplatform.models:endpoint = aiplatform.Endpoint('projects/1075205415941/locations/us-central1/endpoints/7467372802459303936')\n",
      "INFO:google.cloud.aiplatform.models:Deploying model to Endpoint : projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n",
      "INFO:google.cloud.aiplatform.models:Deploy Endpoint model backing LRO: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936/operations/2903536705439006720\n",
      "INFO:google.cloud.aiplatform.models:Endpoint model deployed. Resource name: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n"
     ]
    }
   ],
   "source": [
    "# TODO 3\n",
    "# Deploy the model resource to the serving endpoint resource \n",
    "endpoint = model.deploy(\n",
    "    machine_type=\"e2-standard-4\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fd44380b9ae3"
   },
   "source": [
    "### Predict on the endpoint\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13f3e8aa27c0"
   },
   "source": [
    "* This sample instance is taken from an observation in which `Adopted` = **Yes**\n",
    "* Note that the values are all strings. Since the original data was in CSV format, everything is treated as a string. The transformations you defined when creating your `AutoMLTabularTrainingJob` inform Vertex AI to transform the inputs to their defined types.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "00c0d01dc8ae"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction(predictions=[{'classes': ['Yes', 'No'], 'scores': [0.527707576751709, 0.4722923934459686]}], deployed_model_id='3521401492231684096', explanations=None)\n"
     ]
    }
   ],
   "source": [
    "# TODO 4\n",
    "# Make a prediction using the sample values \n",
    "prediction = endpoint.predict(\n",
    "    [\n",
    "        {\n",
    "            \"Type\": \"Cat\",\n",
    "            \"Age\": \"3\",\n",
    "            \"Breed1\": \"Tabby\",\n",
    "            \"Gender\": \"Male\",\n",
    "            \"Color1\": \"Black\",\n",
    "            \"Color2\": \"White\",\n",
    "            \"MaturitySize\": \"Small\",\n",
    "            \"FurLength\": \"Short\",\n",
    "            \"Vaccinated\": \"No\",\n",
    "            \"Sterilized\": \"No\",\n",
    "            \"Health\": \"Healthy\",\n",
    "            \"Fee\": \"100\",\n",
    "            \"PhotoAmt\": \"2\",\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "print(prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "154258dfb12f"
   },
   "source": [
    "### Undeploy the model\n",
    "\n",
    "To undeploy your `Model` resource from the serving `Endpoint` resource, use the endpoint's `undeploy` method with the following parameter:\n",
    "\n",
    "- `deployed_model_id`: The model deployment identifier returned by the prediction service when the `Model` resource is deployed. You can retrieve the `deployed_model_id` using the prediction object's `deployed_model_id` property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "186856f896fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.models:Undeploying Endpoint model: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.models:Undeploy Endpoint model backing LRO: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936/operations/1845190793006940160\n",
      "INFO:google.cloud.aiplatform.models:Endpoint model undeployed. Resource name: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n"
     ]
    }
   ],
   "source": [
    "# TODO 5 \n",
    "# Undeploy the model resource \n",
    "endpoint.undeploy(deployed_model_id=prediction.deployed_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d7d2aa967f46"
   },
   "source": [
    "# Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial:\n",
    "\n",
    "- Training Job\n",
    "- Model\n",
    "- Endpoint\n",
    "- Cloud Storage Bucket\n",
    "\n",
    "**Note**: You must delete any `Model` resources deployed to the `Endpoint` resource before deleting the `Endpoint` resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "1a9c201f8589"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting AutoMLTabularTrainingJob : projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360\n",
      "INFO:google.cloud.aiplatform.base:Delete AutoMLTabularTrainingJob  backing LRO: projects/1075205415941/locations/us-central1/operations/5317466105709592576\n",
      "INFO:google.cloud.aiplatform.base:AutoMLTabularTrainingJob deleted. . Resource name: projects/1075205415941/locations/us-central1/trainingPipelines/1715908841423503360\n",
      "INFO:google.cloud.aiplatform.base:Deleting Model : projects/1075205415941/locations/us-central1/models/3676687718445744128\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Model  backing LRO: projects/1075205415941/locations/us-central1/operations/8046647479896113152\n",
      "INFO:google.cloud.aiplatform.base:Model deleted. . Resource name: projects/1075205415941/locations/us-central1/models/3676687718445744128\n",
      "INFO:google.cloud.aiplatform.base:Deleting Endpoint : projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n",
      "INFO:google.cloud.aiplatform.base:Delete Endpoint  backing LRO: projects/1075205415941/locations/us-central1/operations/6456876811434328064\n",
      "INFO:google.cloud.aiplatform.base:Endpoint deleted. . Resource name: projects/1075205415941/locations/us-central1/endpoints/7467372802459303936\n"
     ]
    }
   ],
   "source": [
    "delete_training_job = True\n",
    "delete_model = True\n",
    "delete_endpoint = True\n",
    "\n",
    "# Warning: Setting this to true will delete everything in your bucket\n",
    "delete_bucket = False\n",
    "\n",
    "# Delete the training job\n",
    "job.delete()\n",
    "\n",
    "# Delete the model\n",
    "model.delete()\n",
    "\n",
    "# Delete the endpoint\n",
    "endpoint.delete()\n",
    "\n",
    "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
"    ! gcloud storage rm --recursive $BUCKET_NAME"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "automl-tabular-classification.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "name": "tf2-gpu.2-5.m76",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-5:m76"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
